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(57) An apparatus, system, and a method for pro- 
viding more realistic shooting input for shooting video 
games on computers or similar devices is provided. The 
apparatus may be comprised of a mock shooting device, 
such as a gun, having one or two lighting devices con- 
taining a plurality of light sources. The lighting devices 
may use visible or invisible lights. The apparatus may 
also include a screen device for displaying visual target 
objects of a video shooting game, at which a game play- 
er can shoot at with the mock shooting device, a video 
camera that captures video images of the mock shoot- 
ing devices, and a computing device that computes the 



hit position of the mock shooting device on the screen 
device based on the captured video images. The deter- 
mined hit position can then be fed to the video shooting 
game software run by the computing device which can 
determine if a visual target object is actually hit or not, 
and reacts accordingly. The mock shooting device may 
further be comprised of a wireless sender or a wired 
communication line to send command signals from the 
mock shooting device to the computing device. The sys- 
tem and method can be extended to a plurality of game 
players using mock shooting devices with differently 
colored or uniquely spatially arranged lighting devices 
so that a plurality of hit positions may be determined. 
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Description 

Cross Reference to Rolated Applications 

[0001] This present application is a continuation in 
part of and claims the priority of U.S. Patent Application 
No. 1 0,1 28,623, titled "AN APPARATUS AND A METH- 
OD FOR MORE REALISTIC SHOOTING ViDEO 
GAMES ON COMPUTERS OR SIMILAR DEVICES" 
filed on April 23, 2002, inventor and applicant Xiaoling 
Wang. 

Field of the invention 

[0002] This invention relates to the field of systems 
and methods for video games, which entail the use of 
mock shooting devices, such as mock guns. These vid- 
eo games are typically comprised of computer software 
which is run on computers or similar devices. 

Background of the Invention 

[0003] Video games, which entail the use of mock 
shooting devices, are popular and entertaining. These 
video games are typically comprised of computer soft- 
ware which is run on computing devices, such as home 
personal computers. However, most computer video 
games, which entail the use of mock shooting devices 
typically, use computer peripherals, such as a keyboard, 
a mouse or a joystick to aim and shoot at visual targets 
on a computer or video screen. Other similar devices, 
such as the PLAYSTATION (trademarked) from SONY 
(trademarked) and the XBOX (trademarked) from MI- 
CROSOFT (trademarked), use a game pad or other 
game control device to aim and shoot at visual targets 
on a computer video screen. These types of peripheral 
devices make the shooting games somewhat less real- 
istic. 

[0004] There have been some attempts to make video 
games which entail the use of mock shooting devices, 
more realistic. All known prior art in the field of shooting 
video games, as described in the U.S. Patents 
5,366,229 to Suzuki and 6,146,278 to Kobayashi, incor- 
porated herein by reference, rely on three major com- 
ponents: a mock gun that can emit a light beam to a 
target on a screen to be shot at, a video camera that 
photographs the screen for detecting an intersecting 
point of the light beam on the screen, and a position de- 
termination device that determines the actual position 
of the light beam on the screen. The position of the light 
beam on the screen can then be fed back to shooting 
video game control computer software to determine if a 
visual target on a screen is "hit" or not. Some visual and 
audio feedbacksignals indicating hit or miss can be gen- 
erated. Although these systems are more realistic than 
the shooting video games with keyboards or joysticks, 
they are not very suitable for use with the shooting video 
games on computers or similar devices. 



[0005] The main reason is the fact that a normal video 
camera used to photograph a computer monitor screen 
may not be able to provide steady video images of the 
computer monitor screen due to the difference in fre- 
5 quencies of the monitor and the video camera. The mon- 
itor refresh frequency is typically selectable between 
sixty - one hundred and twenty Hz while the video cam- 
era capturing frequency is typically less than thirty Hz. 
The video camera capturing frequency is also process- 
ing speed and image size dependent. Fast computers 
may be able to capture thirty video frames per second 
(thirty Hz) with an image size of 640 by 480 pixels. Slow 
computers may only be able to capture ten frames per 
second (ten Hz) with the same image size and thirty 
frames per second for a smaller size of for example 320 
by 240 pixels. Only if both frequencies are identical or 
the monitor refresh frequency divided by the camera 
capturing frequency is an integer in a more general term, 
steady video images of the monitor screen may be cap- 
tured. Since a computer user may use any refresh fre- 
quency from a wide range of monitor refresh frequen- 
cies and most video cameras have a typical capturing 
frequency of between ten and thirty Hz, it is very com- 
mon that video cameras do not provide steady video im- 
ages from a computer monitor due the frequency mis- 
match. 

[0006] For capturing a steady video from a computer 
monitor, there are only two options. The first option in- 
volves very high-end video cameras with special auto- 
sync functions. Although they can in general provide 
steady video images from most computer monitors, they 
are very expensive. The second option is to preset the 
computer monitor refresh frequency to say sixty Hz or 
seventy-five Hz to keep the needed synchronization 
with the typical video camera with say thirty Hz or twen- 
ty-five Hz, respectively. However, this is only a valid op- 
tion if the same manufacturer ships the video camera 
and the computer monitor used by the shooting video 
game and computer users do not modify the refresh fre- 
quency of their computer monitor. Because video cam- 
eras, computers, monitors and video games in general 
are not from a single vendor or manufacturer and com- 
puter users very often do modify the refresh frequency 
of their monitors, this second option is very often not 
practical. 

[0007] In addition to the frequency mismatch problem 
mentioned above, the camera in the prior art as de- 
scribed in the U.S. Patent No. 5,366,229, incorporated 
by reference herein, must be placed somewhere near a 
game player and facing the same orientation as the 
game player for capturing the display screen. Although 
this may not present a serious problem in a profession- 
ally designed game playing place, it could be very chal- 
lenging to place the video camera at home in such a 
way that it may not easily be occluded at anytime during 
the game and may not easily be bumped into. This is 
not always practical. In order to solve the difficult camera 
placement problem,. the camera as described in the U. 
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S. Patent No. 6,146,278, incorporated herein by refer- 
ence, are integrated with the mock shooting device so 
that the camera is always facing the target screen with- 
out the danger of occlusion. However, this arrangement 
makes the mock shooting device somewhat expensive 
and the integrated video camera totally single-pur- 
posed. Furthermore, the mock shooting device with the 
camera must be connected to the computing device di- 
rectly via a cable, which may also cause inconvenience 
when playing. 

[0008] The above mentioned drawbacks, namely, the 
frequency mismatch between the display screen and 
the low-cost video camera, the difficult placement of the 
video camera facing the screen, relatively high cost for 
a mock shooting device with an integrated camera, as 
well as a needed connection cable between the mock 
shooting device and the computing device, can serious- 
ly limit the applicability of the prior art techniques for 
game players who want to play realistic video shooting 
games with their computers at home. 

Summary of the Invention 

[0009] The present invention in one embodiment 
comprises a computing device; a screen device; and a 
first mock shooting device comprised of one or more 
light sources which are a part of and fixed to the first 
mock shooting device. The computing device is typically 
electrically connected to the screen device. The com- 
puting device uses the light from the one or more light 
sources to determine whether the first mock shooting 
device is aimed towards a first location on the screen 
device. The one or more light sources may flash light 
when the first mock shooting device is triggered. A video 
camera may be used to capture video images of the one 
or more light sources. The video camera may be elec- 
trically connected to the computing device and may pro- 
vide data about the one or more light sources to the com- 
puting device. 

[0010] In at least one embodiment of the present in- 
vention the apparatus is comprised of lighting devices 
using invisible light, such as infrared light, which is only 
invisible to human eyes, but well visible to common vid- 
eo sensors, such as a low-cost web cam. The use of the 
lighting devices with invisible light can effectively elimi- 
nate possible attention distractions of a game player due 
to the flashing lights of the lighting devices with visible 
light. 

[0011] In at least one embodiment of the present in- 
vention the apparatus is comprised of at least three light 
sources fixed to the first mock shooting device and the 
light sources are not located in a single line segment. 
Light sources may include point light sources or area 
light sources. A second mock shooting device com- 
prised of one or more light sources may also be provid- 
ed. Each of the light sources of the first mock shooting 
device may emit light of a first color and each of the light 
sources of the second mock shooting device may emit 



light of a second color, wherein the first color and the 
second color are different. The light sources of the first 
lighting device may have a first spatial arrangement and 
the light sources of the second lighting device may have 

5 a second spatial arrangement, wherein the first spatial 
arrangement and the second spatial arrangement are 
different. Since invisible light, such as infrared light, 
does not have a particular color, a typical video camera 
usually cannot distinguish for example colors from dit- 
to ferent near infrared wavelengths. When lighting devices 
use invisible light, different spatial arrangements for dif- 
ferent mock shooting devices can be necessary for dis- 
tinguishing multiple mock shooting devices. 
[0012] The present invention also includes a method 

*s of using light from one or more light sources fixed to a 
first mock shooting device to determine whether the first 
mock shooting device is aimed towards a first location 
on a screen device. The method may include capturing 
an image of the light through the use of a video camera, 

20 wherein the light can be visible or invisible to humans 
but visible in at least in some embodiments of the 
present invention to the video camera. 
[001 3] The present invention in one or more embodi- 
ments discloses a new system that may use a low-cost 

25 video camera, such as a typical web cam, for capturing 
video images of a mock shooting device instead of a 
computer monitor or a television screen. From the cap- 
tured video images, the pose including position and ori- 
entation of the mock shooting device can be deter- 

30 mined. Please note that pose of an object is defined as 
to include both position and orientation of the object in 
space, as used commonly in thefield of computervision. 
We will use the terminology throughout the present in- 
vention. From the pose of the mock shooting device, the 

35 hit position on the screen or the location on the screen 
towards which the mock shooting device is aimed, can 
be computed. This system can solve the frequency mis- 
match problem and the difficult video camera placement 
problem in the prior art. It also provides a more cost ef- 

40 fective and practical solution for game players using 
their computers or similar devices at home. 
[001 4] The present invention is designed to provide a 
system and a method that can make video games, 
which employ a mock shooting device, much more re- 

45 alistic on computers and/or similar devices. 

[0015] A system, apparatus, and a method according 
to the present invention uses a mock shooting device, 
such as a mock gun, a mock machine gun, or a mock 
rocket launcher, with two lighting devices, a position 

50 lighting device and a command lighting device, each 
containing one or more light sources. A game player us- 
es the mock shooting device with the position lighting 
device being turned on to aim at one of one or more 
target objects displayed on a screen by a video shooting 

55 game. A typical low-cost video camera captures video 
images containing images of the light emitted from the 
light sources from the position lighting device fixed to 
and/or apart of the mock shooting device. The hit posi- 
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tlon at which the mock shooting device aims (points to) 
can quickly be determined from the captured video im- 
ages. The determined hit position is then passed to the 
video shooting game software running on the computing 
device that displays a target cursor at the hit position on 
a screen device. When the game player moves the 
mock shooting device, the determined hit position rep- 
resented by the target cursor on the screen, such as a 
cross hair cursor, moves accordingly. This gives the 
game player immediate feedback about where his/her 
mock shooting device is pointing to and allows the play- 
er to plan how the mock shooting device should be 
moved to get on top of the target object of interest. When 
the mock shooting device is triggered, the command 
lighting device of the mock shooting device may flash 
light to signal a shooting command. The triggered hit po- 
sition can then be fed to the video shooting game soft- 
ware that determines if a visual target is "hit" or not, and 
can react accordingly. 

[001 6] A system, apparatus, and a method according 
to the present invention uses a mock shooting device 
with an integrated control pad. The integrated control 
pad is comprised of a plurality of control buttons com- 
monly available to the control pads for Xbox (trade- 
marked) or PlayStations (trademarked). The integrated 
control pad is used to send control commands (signals), 
such as different weapon selection commands or shoot- 
er's positioning, among many other commands. The 
commands of the control pad can be sent from the mock 
shooting device to the computing device in the following 
three ways: 

(1 ) The control commands can be encoded and sent 
via a lighting device showing different lighting pat- 
terns or colors or a combination of both. The lighting 
patterns or colors can then be captured by the video 
camera. The video image containing the light pat- 
terns can then be analyzed and decoded as com- 
mands by the computing device. 

(2) A wireless connection can be used to send the 
control signals from the mock shooting device to the 
computing device. The wireless connection can be 
radio-wave based, such as the connections of a 
cordless telephone between its base station and its 
handset or a remote garage door opener, or infrared 
light based, such as a common remote control for 
home electronics. This embodiment typically would 
require a receiving device to be placed and con- 
nected to the computing device. 

(3) A wired connection can also be used to send the 
control signals from the mock shooting device to the 
computing device. The wired connection can be the 
same as the wired connections between a game 
console and its control pad. Although a wired con- 
nection is simpler and cheaper than a wireless con- 
nection, it does constrain the freedom of movement 
of a game player. 
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[0017] The system, apparatus, and method in accord- 
ance with embodiments of the present invention offer 
the following advantages: 

s (1 ) The video camera needed for the system can be 
a general-purpose, low cost video camera that can 
be used for many other applications, such as video- 
conferencing. 

(2) A game player may be able to use his/her exist- 
to ing web cam for playing the shooting game. 

(3) The mock shooting device according to some 
embodiments of the present invention does not 
need a cable to connect to the computing device. 
This lack of need for a connection cable imposes 

*5 less movement constraints and provides a greater 
possible shooting distance range for the mock 
shooting device. 

(4) The mock shooting device needed for the sys- 
tem can also be a multi-purpose low cost device. 

20 The mock shooting device, such as a mock machine 
gun, can be used and played just like a toy machine 
gun with or without the shooting game. 

Brief Description of the Drawings 

25 

[0018] 

Fig. 1 is a perspective view schematically illustrat- 
ing the overall structure of the preferred embodi- 
30 ment of the present invention; 

Figs. 2 A and 2B illustrate point and area light sourc- 
es shown in video images; 
Fig. 3 is a block diagram or flow chart schematically 
illustrating a method which can be executed by a hit 
35 position determination device and a command de- 
termination device; 

Fig. 4 is a perspective view schematically illustrat- 
ing the overall structure of another embodiment of 
the present invention; 
to Rg. 5 is a perspective view schematically illustrat- 
ing the overall structure of another embodiment of 
the present invention with a wireless connection for 
sending commands; and 

Fig. 6 is a perspective view schematically illustrat- 
es ing the overall structure of another embodiment of 
the present invention with a wired connection for 
sending commands. 

Detailed Description of the Invention 

50 

[001 9] The present invention in one or more embodi- 
ments provides a solution that can make shooting video 
games much more realistic on computers or similar de- 
vices, such as the PLAYSTATION (trademarked) from 
55 SONY (trademarked), that contain at least one proces- 
sor, amemory device and/or a storage device, a monitor 
or a display screen, such as a television set, a low cost 
video camera, and some input devices, such as a game 
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pad, and/or joysticks. 

[0020] A system, apparatus, and method according to 
the present invention uses a mock shooting device, 
such as a mock gun, a mock machine gun, or a mock 
rocket launcher, with a position lighting device and a 
command lighting device. A game player uses the mock 
shooting device with the position lighting device being 
turned on to aim at one of one or more target objects 
displayed on a screen by a video shooting game. When 
the mock shooting device is triggered, the command 
lighting device flashes light to send a shooting command 
signal to the computing device. Both lighting devices in- 
clude one or more light sources and are mounted on or 
built in the mock shooting device. The mock shooting 
device can be triggered continuously with a predefined 
time interval when its triggering device is pulled back 
and not released or the mock shooting device can be 
triggered just one time with a quick pull back and re- 
lease. The mock shooting device may also provide au- 
dio or visual feedback signals indicating that the device 
has been triggered. For example, the mock shooting de- 
vice may play a very short and typical gun shooting 
sound clip when it is triggered. When it is continuously 
triggered, the very short and typical gun shooting sound 
clip will be repeated with a predefined time interval as 
long as the trigger is pulled back and not released. 
[0021] A system, apparatus, and method according to 
the present invention uses a commonly available low- 
cost video camera, such as a web cam, mounted on top 
of a screen device, such as a computer monitor or a TV 
set, to capture the video images containing the light from 
the lighting devices. The hit position on the screen at 
which the mock shooting device aims and shoots, can 
be determined from the captured video images contain- 
ing the mock shooting device with the lighting devices. 
The hit position can then be fed to the shooting video 
game computer software which can determine if a target 
is actually hit or not. It should be noted that hereinafter 
the word "hit", used throughout this application, is meant 
to be a virtual hit on the screen by a virtual bullet fired 
by the mock shooting device, instead of an actual hit in 
a physical sense. 

[0022] A perspective view of a system, apparatus, 
and method according to one preferred embodiment of 
the present invention is shown in Fig. 1 . Fig. 1 shows 
an apparatus 1 00 comprised of a mock shooting device 
110, a screen device 130, a video camera 150, and a 
computing device 1 70. The computing device 1 70 may 
be a personal computer or a game console machine, or 
other similar devices. The screen device 130 is electri- 
cally connected to the computing device 170 by com- 
munications line 170a. The video camera 150 is electri- 
cally connected to the computing device 170 by com- 
munications line 150a. The communications lines 150a 
and 1 70a may be comprised of wireless connections, 
hardwired connections, optical connections, software 
connections, or any other known communication con- 
nections. 



[0023] The mocking shooting device 110 includes a 
position lighting device 1 1 5 and a command lighting de- 
vice 116. The position lighting device 115 may be com- 
prised of three lights 115a, 115b, and 115c. The screen 

5 device 1 30 can display target visual objects to be aimed 
and shot at. The video camera 150 may be used to cap- 
ture video images from the mock shooting device 110 
and the video camera 150 can be mounted onto the 
screen device 130. The computing device 170 may be 

10 comprised of a hit determination device 1 80, which may 
be comprised of computer software which is part of and 
is running on the computing device 170. The hit deter- 
mination device 180 may determine the hit position, 
such as hit position 131, on the screen device 130 at 

is which the mock shooting device 110 was aiming and 
shooting. 

[0024] The shooting path (trajectory) 110a is the vir- 
tual shooting path of a virtual bullet from the mock shoot- 
ing device 110 to the screen device 130. The light from 

20 lights 115a, 115b, 115c, and 116a or some other light is 
usually non-directional so that they can be observed 
from a large range of directions. For this reason, each 
of lights 115a-c and 116a may be a typical small light 
bulb or a small LED. The lights 1 1 5a-c and 1 1 6a do not 

25 need to be expensive directional lights, such as lasers. 
The screen device 1 30 includes a screen 130a on which 
visual target objects, such as target object 132, are dis- 
played. The computing device 170 is also responsible 
for running the shooting game 190, which may be corn- 
so prised of computer software, that displays visual target 
objects to be shot at on the screen 130a and reacts ac- 
cordingly depending on whether a visual target object 
has been hit or not. With some exceptions, the video 
shooting game 1 90 may be similar to those prior art vid- 

35 eo shooting games which are typically comprised of 
computer software and which run on computers. One of 
the differences of the present invention is how user 
shooting information is input into the computing device 
170. The system and method according to the present 

40 invention uses a realistic mock shooting device 1 1 0 and 
a video camera 1 50 for inputting user shooting informa- 
tion while conventional prior art games use a keyboard, 
mouse, game pad or joysticks. 
[0025] In operation, referring to Fig. 1 , a game player 

45 starts a video shooting game 1 90 stored in the comput- 
ing device 170. The video shooting game 190 may be 
initially supplied to the computing device 1 70 via com- 
pact disc, floppy disc, downloaded from the Internet, or 
in any other known manner. The shooting game 1 90 dis- 

50 plays scenes with one or more visual target objects, 
such as circular target object 132, on the screen 130a 
via communications line 1 70a. Typical examples of the 
communications line 170a are common video display 
cable and the Universal Serial Bus (USB) cable version 

55 1 ,1 and 2.0 for computer monitors, and composite video, 
S-video, or RGB (red, green, blue) video cables for tel- 
evision sets. The game player uses the mock shooting 
device 1 1 0 to aim and shoot at the displayed target ob- 
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jects provided by the video shooting game 190 on the 
screen 130a. After the game player starts the game, the 
light sources 115a-c of the position lighting device 115 
will be tuned on using the switch 1 1 8. The light sources 
1 15a-c are each rigidly mounted on or integrated within 
the mock shooting device 110. The video camera 150 
placed on top of the screen device 130 captures video 
images from the light sources 115a-c of the position 
lighting device 1 1 5 and sends the video images through 
communications line 1 50a to the computing device 1 70. 
Typical and common examples of the communications 
line 150a are the Universal Serial Bus (USB) cable ver- 
sion 1.1 and 2.0, or cables made according to the IEEE 
1394 standard, such as the FIREWIRE (Trademarked) 
and the ILINK (T rademarked copyrighted). The hit posi- 
tion determination device 1 80 running on the computing 
device 170 then processes the captured video images. 
The hit position determination device 1 80 computes the 
position and the orientation of the lighting device 115 
based on the positions of the plurality of light sources 
115a-c of the lighting device 115 in the video images. 
The position and the orientation of the mock shooting 
device 110 can then be determined since the lighting 
device 115 has a known and fixed spatial relationship 
with respect to the mock shooting device 110. (This as- 
sumption is always valid since we require that the posi- 
tion lighting device 115 is either an integral part of or 
rigidly mounted on the mock shooting device 110). 
Based on the computed position and the orientation of 
the mock shooting device 1 1 0 relative to the screen 
130a, the hit position of the virtual bullet from the mock 
shooting device 110 on the screen 130a can finally be 
calculated. The hit position is then passed to the video 
shooting game 190 running on computing device 170. 
By using the hit position information, a target cursor is 
displayed on the screen 130a so that the game player 
can see where exactly is his/her shooting device point- 
ing to and determine how it should be moved to hit a 
target object of interest. When the target cursor is right 
on top of the target object, the game player can trigger 
the mock shooting device 110 to shoot at the target. 
When the mock shooting device 110 is triggered, the 
light source 116a of the command lighting device 116 
flashes light to signal a shooting command. The shoot- 
ing command is captured by the camera 150, and the 
camera 1 50 feeds the captured image to the computing 
device 170, and then the computing device 170 deter- 
mines the command signal from the image containing 
the light sources from the command lighting device and 
sends it further to the video shooting game 1 90. The vid- 
eo shooting game 1 90 running on the computing device 
1 70, determines whether an actual visual target object, 
such a target object 1 32, has been h it or not by the virtual 
bullet from mock shooting device 1 1 0 and if so the shoot- 
ing game 190 executes the appropriate computer soft- 
ware instructions for a "hit" scenario. 
[0026] The position and the orientation of the mock 
shooting device 110 in space can be determined indi- 



rectly via a pose estimation of the rigidly mounted or in- 
tegrated position lighting device 1 1 5. This indirect meth- 
od reduces the computational complexity and improves 
the robustness of the method significantly. The advan- 
5 tages can be summarized as follows: 

(1) No difficult object and background separation 
problem. The pose estimation of a general three- 
dimensional object, such as the mock shooting de- 

10 vice 110, is not always simple, when the object is 
not easily separable from the background or the en- 
vironment in which the object exists. The object and 
background separation problem in general is re- 
garded as a difficult computer vision problem that 

15 is not always easily solvable. However, if the light 
sources 115a-c of the position lighting device 115 
have been turned on, the light sources will be im- 
aged as bright blobs in video images. When the 
mock shooting device 110 is triggered, the light 

20 source 116a of the command lighting device 116 
that flashes light will also be imaged as an additional 
bright blob in video images captured by the video 
camera 150. Bright blobs are in general very easily 
detectable and hence quickly separable from a 

25 background without additional bright light sources. 
This assumed condition is usually not difficult to be 
satisfied. 

(2) Low localization complexity of feature points. 
For object pose estimation, object feature points, 

30 such as edge, junction and corner points, must be 
localized. In general, these image feature points 
take longer to compute than the detection of simple 
bright blobs generated by point or area light sourc- 
es. 

35 (3) Furthermore, bright blobs can be detected much 
more reliably than common image feature points. 
This is especially true if the image contrast is low 
and the noise level is high, when the image is taken 
under a typical low illumination condition. 

40 

[0027] As discussed above, the position lighting de- 
vice 115 plays a significant role for solving the pose es- 
timation of the mock shooting device 110. There is a 
question of how many points do we need to estimate the 

45 pose of the mock shooting device 110 via the position 
lighting device 115. As stated in the reference by Ml. 
Liu et. al., which is incorporated by reference herein, 
three non-collinear corresponding points (i.e. three im- 
age points that are not arranged along a single line in 

50 space) are sufficient for the pose estimation of an object. 
However, in order to make the pose estimation more re- 
liable, four or more points may be helpful. For example, 
a method with four points js proposed in the reference 
by M.L. Liu et. al.. The proposed method works with four 

55 non-collinear (i.e. all points are not arranged along a sin- 
gle line in space) points that can either be co-planar (i. 
e. all points are arranged along a single plane in space) 
or non-coplanar (i.e. all points are not arranged along a 
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single plane in space). The proposed method may also 
be extended to handle more points. Because the pose 
estimation problem with image points is a well-known 
and solved problem, details will not be described in this 
invention and can be found in the cited reference. It is 
important to point out that the cited reference only 
serves the purpose of a common reference, it does not 
indicate in any way that the method is the preferred one, 
but only that it can be used with the system and the 
method according to one or more embodiments of the 
present invention. Therefore, we can conclude that a 
minimum of three non-collinear point light sources, such 
as 1 1 5a, 1 1 5b, and 1 1 5c shown in Fig. 1 , should be used 
for the position lighting device 115 for pose estimation 
with reasonable accuracy. For better accuracy, four or 
more non-collinear point light sources may be used. If 
high accuracy is not required, less than three points may 
also be used for providing a rough estimation of the hit 
position. 

[0028] While the position lighting device 115 is de- 
signed for the pose estimation, the command lighting 
device 116 is for sending command signals. The com- 
mand lighting device 116 as shown in Fig. 1 contains 
only one light source 1 1 6a because only one command 
signal, the shooting command in this case, needs to be 
sent. When more command signals are needed, more 
light sources may be needed for the command lighting 
device. The use of more than one light source for a com- 
mand lighting device, similar to device 116 will be dis- 
cussed later. 

[0029] There are two common types of light sources, 
which may be used for solving our pose estimation. A 
point light source is a light source with a very small and 
isolated, most likely rounded lighting area that repre- 
sents only a few bright pixels or a very small bright spot 
in a video image. Typical examples of point light sources 
in a video image are shown and marked as point light 
sources 315a-315c in video image 316 in Fig. 2A. The 
position of a point light source in a video image can eas- 
ily be localized through determining the position of the 
centroid of a smalt and isolated bright blob. For a point 
light source, the shape of a point light source, such as 
point light source 31 5a, is normally not used or evaluat- 
ed for pose estimation due to its compact size. As men- 
tioned previously, we typically need at least three point 
light sources for estimating the pose of the mock shoot- 
ing device 1 1 0. In contrast, for an area light source, such 
as a light source in the shape of a triangle or a rectangle, 
such as triangular light source 215 in video image 216 
in Fig. 2A and rectangular light source 415 in video im- 
age 41 6 shown in Fig. 2B, respectively, the light source's 
shape may be used for computing the position and the 
orientation of the light source. In general, one area light 
source with, say three or four, corners, can be seen as 
equivalent to three or four point light sources, respec- 
tively. As shown in Fig.2A, for example, the three corner 
points, 215a-c, of a triangular-shaped area light source 
215 can easily be extracted and these three extracted 



comer points can be viewed as similar to the three point 
light sources 315a-c, arranged in a triangular shape. 
Similarly, a rectangular area light source 415, shown in 
Fig. 2B, has four corner points, 415a-d, that can be seen 
5 as or equivalent to four co-planar point light sources 
515a-d. 

[0030] Therefore, one triangular area light source 
may be sufficient to satisfy the minimum condition of 
three point light sources for the pose estimation, as 

10 mentioned previously. Depending on the design of the 
mock shooting device 110, the lighting device 115 may 
be comprised of point light sources, area light sources, 
or a combination of both. In general, more light sources 
lead to more accurate and robust pose estimation. How- 

*s ever, on the other hand, more light sources mean longer 
computational time and higher production cost. 
[0031] Fig. 3 shows a flow chart 500 illustrating a 
method which can be executed by a hit position deter- 
mination device running on computing device 1 70, such 

20 as device 180 shown in Fig. 1 , for determining the hit 
position of a virtual bullet shot from the mock shooting 
device 11 0. At step 51 0 a video image is captured. The 
video image may be captured by video camera 150, 
which then transmits data via the communications line 

25 1 50a to the computing device 1 70. The captured video 
image may be subjected to a bright blob localization 
process by hit position determination device 1 80 at step 
530. The computing device 170, which runs the hit de- 
termination device 1 80 computer software, may scan 

30 through the whole captured video image pixel by pixel 
and may compare a pixel intensity value with a given or 
computed threshold value which may be stored in mem- 
ory of the computing device 170. Pixels with intensity 
value greater than the threshold value may be identified 

35 as "bright" pixels by the computing device 1 70. If the 
computing device 170 cannot find any bright pixels in 
the image, the computing device 1 70 determines that 
no lighting device of the mock shooting device 110 was 
turned on when the captured video image was captured 

40 and no further processing is needed. Otherwise, the 
computing device 1 70 determines if the detected bright 
pixels form bright blobs with bright neighboring pixels. 
This step 530 essentially removes noisy pixels and lo- 
calizes the bright blobs. The identified bright blobs are 

45 then compared with a given expected size range of the 
bright blobs as well as the given expected total number 
of bright blobs for verifying the correctness of the blob 
localization. For example, if a system uses three point 
light sources in its position lighting device and one point 

so light source in its command lighting device, and the blob 
size of each imaged point light source is between five 
and ten pixels in diameter, the computing device 1 70 will 
check if the total number of bright blobs is four or three 
(corresponding to with or without the command lighting 

55 device being turned on) and if the diameter of each 
bright blob is indeed between five and ten pixels. Only 
if both checks are successful, the computing device 1 70 
can be certain. that the localized bright blobs are indeed 
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coming from the four orthree expected point light sourc- 
es. Otherwise, the computing device 1 70 may decide to 
go back to look for more bright blobs in the image with 
a lowered threshold value or post an error message. The 
localized bright blobs are then subjected to a position 
determination process at step 540 by the computing de- 
vice 170 for blob center and blob corners. 
[0032] If only point light sources are used in the light- 
ing device, the computing device 170 at step 540 will 
perform position determination for each blob center. The 
center position of a blob can easily be computed by av- 
eraging the pixel coordinates of each pixel within the 
blob. If one or more area light sources are used, the 
computing device 170 at step 540 will perform corner 
detection for every given bright blob. For example, if one 
rectangular-shaped area light source is used in the light- 
ing device, the computing device 170 will localize four 
expected corners. Since corner detection methods are 
very common and basic in the computer vision field and 
described in almost all textbooks about computervision 
and image processing, we skip the details for simplicity 
and clarity of the description. When a mixture of point 
and area light sources are used, both blob center and 
corner detections are needed. 
[0033] The localized center and/or comer points are 
then passed to a blob separation process at step 545. 
At step 545, the computing device 170 separates the 
blobs corresponding to the position lighting device 115 
and the command lighting device 116 by comparing of 
the blob positions with the known geometry of both light- 
ing devices. 

[0034] After the blob separation, the blobs corre- 
sponding to the position lighting device 115, have been 
separated out at step 547, and will be fed to a pose es- 
timation process at step 550. At step 550, the computing 
device 170 takes center and/or comer points as input, 
and estimates the position and the orientation of the 
lighting device 115. This method works with either point 
or area light sources. The type of light sources makes 
only differences in step 540 as discussed in the previous 
section: locate blob center points for point light sources 
and locate blob corner points for area light sources. A 
good working method for pose estimation with four fea- 
ture points is well described in the reference by M.L. Liu 
et al., which is incorporated by reference herein. Since 
there are many published pose estimation methods that 
could be used with the present invention without modi- 
fication, and the description of the pose estimation proc- 
ess itself is complicated, the present application again 
skips the details. After the pose (position and orienta- 
tion) of the mock shooting device 110 is determined by 
the computing device 1 70 at step 550, the shooting path 
1 1 0a of the virtual bullet from the mock shooting device 
110 as shown in Fig. 1 can easily be obtained. 
[0035] Once the shooting path 110a of the virtual bul- 
let from the mock shooting device 110 is computed by 
the pose estimation process at step 550, the hit position 
of a virtual bullet from the mock shooting device 1 1 0 can 



then be computed by the computing device 1 70 by a hit 
position estimation process at step 560 shown in Fig. 3. 
The hit position estimation process at step 560 treats 
the display screen 130a as a plane with its known posh 
5 tion and orientation and the shooting path 110a or line 
of the mock shooting device 110 as a line in space with 
its known position and orientation, and computes the in- . 
tersection point of the plane (i.e. display screen 130a) 
and the line (i.e. shooting path or line 110a). The inter- 
section point is the hit position, such as position 131 , of 
the virtual bullet on the display screen 1 30a. The hit po- 
sition estimation process at step 560, executed by com- 
puting device 1 70, finally outputs the screen hit position 
at step 590 to the shooting game 1 90 which is computer 
software running on the computing device 170. 
[0036] In parallel to the pose estimation for the blobs 
from the position lighting device 115, the blobs from the 
command lighting device 116, determined at step 548, 
are fed to the command determination process at step 
562. When only one point light source is used in the 
command lighting device 1 1 6 as in the current example, 
the computing device 1 70 at step 562 simply determines 
if there is a bright blob from the command lighting device 
116 in the image. If this one bright blob has been found, 
this process simply outputs the determined shooting 
command, such as "fired" at step 592. 
[0037] Finally, the video shooting game 1 90 uses the 
outputs supplied at steps 590 and 592 as input to deter- 
mine if an actual visual target object displayed on the 
display screen 130a has been hit or not and the video 
shooting game 1 90 reacts depending on how it is pro- 
grammed. 

[0038] The apparatus 1 00 shown in Fig. 1 may be ex- 
tended to include a plurality of mock shooting devices, 
each of which may be identical to the mock shooting de- 
vice 110 equipped with lighting devices 115 and 116 us- 
ing different colors for multiple game players. If the video 
camera 150 is a color camera, light sources in different 
colors can easily be distinguished. For example, for a 
dual user apparatus, two mock shooting devices, each 
like 110, one mock shooting device having only red light 
sources, such as one or more red light sources of a red 
lighting device and one mock shooting device having 
only green light sources such as one or more green light 
sources of a green lighting device, may be operated by 
two game players. The pose of the two mock shooting 
devices may be determined separately by locating the 
red pixels for one of the mock shooting devices and the 
green pixels for the other in the same video images as 
long as the red pixels and the green pixels are not over- 
lapping in space. When an overlap of one or more of the 
lighting sources of the lighting devices occurs, some red 
or green light sources may be occluded and hence no 
longer detectable. This may lead to inaccurate pose es- 
timation and in the worst case to wrong pose estimation 
results. Therefore, if more than one player are playing, 
it is important to keep a certain minimum distance be- 
tween all mock shooting devices for accurate pose es- 
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timation of the mock shooting devices. 
[0039] When invisible light is used by the lighting de- 
vices, no color separation is possible. In this case, each 
of the mock shooting devices should have its own char- 
acteristics for easy differentiation. For example, a mock 
shooting device may use point light sources arranged 
in a triangular shape, while others may contain point 
light sources arranged in a rectangular shape or in a 
more general polygonal shape. Furthermore, if one 
mock shooting device contains only point light sources, 
the others may be comprised of area light sources or a 
combination of point and area light sources. In general, 
the characteristics of each lighting device, such as its 
shape and spatial distribution, should be as different as 
possible for easy separation. 
[0040] There are two main types of video shooting 
games available on the market today. The first type dis- 
plays only targets on a screen to be shot at. The main 
task for a game player playing with this type of shooting 
game is to shoot and destroy the displayed targets. The 
present invention in various embodiments can serve this 
type of video shooting game very well by making them 
more exciting and realistic. The second type of video 
shooting games displays not only targets , to be de- 
stroyed but also a shooter on a screen who can actively 
seek and destroy the targets. Furthermore, a game play- 
er may also change his/her weapon dynamically de- 
pending on needs. The game player may for example 
select to use a simple handgun in some cases, a ma- 
chine gun when more enemies are present, a rifle with 
a telescope to play like a sniper, or even an anti-tank 
missile launcher for destroying enemy tanks. For this 
type of more advanced video shooting games, it is cer- 
tainly desirable if a mock shooting device can not only 
reveal its pose and send a simple shooting command, 
but also send more advanced commands mentioned 
previously. Fortunately, only small modifications of the 
mock shooting device 1 1 0 are needed to make this pos- 
sible, as shown in Fig. 4. Fig. 4 shows apparatus 600 
comprised of a mock shooting device 61 0, a screen de- 
vice 630, a video camera 650, and a computing device 
670. The computing device 670 may be a personal com- 
puter or a similar device. The screen device 630 is elec- 
trically connected to the computing device 670 by a 
communications line 670a. The video camera 650 is 
electrically connected to the computing device 670 by 
communications line 650a. The communications lines 
650a and 670a may be comprised of wireless connec- 
tions, hardwired connections, optical connections, soft- 
ware connections, or any other known communication 
connections. Devices 610, 630, 650, and 670 of appa- 
ratus 600 shown in Fig. 4 are similar to the devices 110, 
130, 150 and 170 of apparatus 100 shown in Fig. 1. In 
comparison with the apparatus 100 shown in Fig. 1 , the 
apparatus 600 has mainly two differences between the 
mocking shooting device 61 0 and the mock shooting de- 
vice 110. The position lighting device 615 is similar to 
the position lighting device 115 shown in Fig. 1 . Howev- 



er, unlike the command lighting device 116, the com- 
mand lighting device 616 is comprised of a plurality of 
light sources instead of one, such as light sources 61 6a- 
d. The command lighting device 616 may be rigidly 

5 mounted to the device 610. While the position lighting 
device 615 is activated by a separate switch 618 of the 
mock shooting device 610, similar to mock shooting de- 
vice 1 1 0, the command lighting device 61 6 is controlled 
by the trigger 612 and a plurality of control buttons of an 

10 integrated control pad, such as buttons 619a-g. The 
control buttons 61 9a-g may be placed anywhere on the 
mock shooting device 61 0 as long as the buttons 61 9a- 
g can easily be reached and operated by the thumbs or 
fingers of a game player. The placement of the control 

*5 buttons 619a-g shown in Fig. 4 serves only as one ex- 
ample for the simple illustration purpose. More ergo- 
nomic designs may be employed to make the control 
buttons more accessible and more efficient. When the 
position lighting device 61 5 is turned "ON" by the switch 

20 618, the light sources 615a-c can easily be imaged as 
bright blobs by the video camera 650. The computing 
device 670 performs pose estimation processing steps 
similar to computing device 1 70, to determining the pose 
of the position lighting device 615. Since the position 

25 lighting device 615 is typically rigidly mounted on or an 
integrated part of the mock shooting device 610, the 
pose of the . mock shooting device 610 can be deter- 
mined. Independently, the trigger 61 2 of the mock shoot- 
ing device 610 controls the command lighting device 

30 61 6. When the trigger 61 2 is pulled back, the command 
lighting device 616, including light sources 616a-d, 
flashes one or more lights to send the shooting com- 
mand signal, independent of the state of the position 
lighting device 615. When each of the control pad but- 

35 tons 61 9a-g is operated by a game player, the command 
lighting device 616 will flash a different and a unique 
combination of lights of the light sources 616a-d. Since 
four lights in the current example can theoretically cre- 
ate sixteen unique combinations (four lights, each with 

40 two states "ON" and "OFF", are equivalent to an encode 
with four bits that can encode sixteen combinations), up 
to sixteen command signals can be sent. The shooting 
command can be one of the command signals to be en- 
coded. The trigger 61 2 can be seen as a special control 

45 button that is always associated with the shooting com- 
mand. Other control buttons are more flexible and can 
be associated with any commands or command signals 
of interest. In general, the more command signals that 
need to be sent, the more lights that are needed for the 

so command lighting device 616. For great robustness of 
the command signal detection with for example error 
correction capabilities, more lights may be used for 
sending command signals with a certain degree of re- 
dundancy. 

55 [0041] A perspective view of a system, apparatus, 
and method according to another embodiment of the 
present invention is shown in Fig. 5. Fig. 5 shows an 
apparatus 700 comprised of a mock shooting device 



10 



17 



EP 1 435 258 A2 



18 



710, a screen device 730, a video camera 750, and a 
computing device 770. The computing device 770 may 
be a personal computer or a similar device. The screen 
device 730 is electrically connected to the computing 
device 770 by communications line 770a. The video 
camera 750 is electrically connected to the computing 
device 770 by communications line 750a. The commu- 
nications lines 750a and 770a may be comprised of 
wireless connections, hardwired connections, optical 
connections, software connections, or any other known 
communication connections. Devices 710, 730, 750, 
and 770 of the apparatus 700 shown in Fig. 5 are similar 
to the devices 610, 630, 650 and 670 of the apparatus 
600 shown in Fig. 4. In comparison with the apparatus 
600 shown in Fig. 4, the apparatus 700 has the following 
two differences: 

(1) The command lighting device 61 6 shown in Fig. 
4 has been replaced with a wireless sender 720 as 
shown in Fig. 5. All control command signals are 
now encoded in the embodiment of Fig. 5, and sent 
via the wireless sender 720 instead of the command 
lighting device 616 in Fig. 4. 

(2) A wireless receiver 740 is now added to the ap- 
paratus for receiving the control command signals 
sent by the wireless sender 720. The wireless re- 
ceiver 740 communicates with the computing de- 
vice 770 via the communications line 740a. The 
communications line 740a can be eliminated when 
the wireless receiver 740 is a built-in component of 
the computing device 770. 

[0042] The wireless connection with the sender 720 
and the receiver 740 can be radio wave based or infra- 
red light based. Cordless phones and remote garage 
door openers are good examples of simple and low-cost 
wireless systems based on radio waves. Common re- 
mote controls used to control home electronics, such as 
TVs and VCRs, are mostly based on infrared light. In 
general, radio wave and infrared light are proven and 
low cost technologies. Both can be used in the embod- 
iments according to the present invention. 
[0043] The apparatus 700 may further be simplified 
by integrating the wireless receiver 740 into the video 
camera device 750. By doing so, the apparatus 700 has 
one less component and one less connection line 740a 
with the computing device 770. This can make the whole 
system cheaper and compacter with less messy cables. 
The main disadvantage of this integrated solution is the 
fact that ordinary web cams can no longer be used with 
such a system. 

A perspective view of a system, apparatus, and method 
according to another embodiment of the present inven- 
tion is shown in Fig. 6. Fig. 6 shows apparatus 800 com- 
prised of a mock shooting device 810, a screen device 
830, a video camera 850, and a computing device 870. 
The computing device 870 may be a personal computer 
or a similar device. The screen device 830 is electrically 



connected to the computing device 870 by communica- 
tions line 870a. The video camera 850 is electrically con- 
nected to the computing device 870 by communications 
line 850a. The communications lines 850a and 870a 

5 may be comprised of wireless connections, hardwired 
connections, optical connections, software connec- 
tions, or any other known communication connections. 
Devices 81 0, 830, 850, and 870 of apparatus 800 shown 
in Fig. 6 are similar to the devices 710, 730, 750 and 

10 770 of apparatus 700 shown in Fig. 5. In comparison 
with the apparatus 700 shown in Fig. 5, the apparatus 
800 has the following main differences: The wireless 
connection including the sender 720 and the receiver 
740 are now replaced by a wired communications line 

is 81 0b. All control command signals are now sent in the 
embodiment of Fig. 6, and received via the communica- 
tions line 81 Ob. The main advantage of this approach is 
its low cost and simple implementation. The main draw- 
back is the constraint of movement freedom of a game 

20 player caused by such a wired cable between the mock 
shooting device 81 0 and the computing device 870. 
The apparatus 700 shown in Fig. 5 and the apparatus 
800 shown in Fig. 8 may also be extended to include a 
plurality of mock shooting devices, each of which may 

25 be identical to the mock shooting device 110 equipped 
with lighting devices 115 and each of which uses a dif- 
ferent characteristic, such as a different color or a dif- 
ferent spatial arrangement, or a combination of them, 
for multiple players, in addition, each of the plurality of 

30 mock shooting devices may also have a unique identi- 
fier. The unique identifier may be just a unique identifi- 
cation number or other unique properties. Each of the 
plurality of mock shooting devices may use its unique 
identifier to send command signals so that the comput- 
es ing device can easily distinguish command signals from 
each of the plurality of mock shooting devices. Although 
the invention has been described by reference to par- 
ticular illustrative embodiments thereof, many changes 
and modifications of the invention may become appar- 

^o ent to those skilled in the art without departing from the 
spirit and scope of the invention. It is therefore intended 
to include within this patent all such changes and mod- 
ifications as may reasonably and properly be included 
within the scope of the present invention's contribution 

45 to the art. 



Claims 



so 1. An apparatus comprising 
a computing device; 
a screen device; 

a first mock shooting device having a position light- 
ing device comprised of one or more light sources 
55 which are fixed to the mock shooting device; and 

wherein the computing device is electrically 
connected to the screen device; and 

wherein the computing device uses the light 
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from the one or more light sources of the position 
lighting device to determine whether the first mock 
shooting device is aimed towards a first location on 
the screen device; and 

wherein the first mock shooting device is fur- 
ther comprised of a command lighting device which 
is comprised of one or more light sources; and 

wherein the one or more light sources flash 
light when the first mock shooting device is trig- 
gered. 

2. The apparatus of claim 1 further comprising 

a video camera that captures video images of the 
one or more light sources; 

wherein the video camera is electrically con- 
nected to the computing device and provides data 
about the one or more light sources to the comput- 
ing device. 

3. The apparatus of claim 1 wherein 

the light from the one or more light sources of the 
position lighting device is invisible to human eyes 
but visible to the video camera. 

4. The apparatus of claim 1 further comprising 

a second mock shooting device comprised of a po- 
sition lighting devices; 

wherein the position lighting device of the sec- 
ond marking device is comprised of one or more 
light sources; 

wherein each of the one or more light sources 
of the position lighting device of the first mock shoot- 
ing device has a first characteristic; 

wherein each of the one or more light sources 
of the position lighting device of the second marking 
device has a second characteristic; and 

wherein the first characteristic and the second 
characteristic are different. 

5. A method comprising the steps of 

using light from one or more light sources of a po- 
sition lighting device fixed to a first mock shooting 
device to determine whether the first mock shooting 
device is aimed towards a first location on a screen 
device. 

6. The method of claim 5 further comprising 
capturing an image of the light through the use of a 
video camera. 

7. The method of claim 5 wherein 

the light from at least two light sources of the posi- 
tion lighting device is used to determine whether the 
first mock shooting device is aimed towards the first 
location on the screen device. 

8. The method of claim 5 further comprising 

using light from one or more light sources of a po- 
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sition lighting device fixed to a second mock shoot- 
ing device to determine whether the second mock 
shooting device is aimed towards a second location 
on a screen device; and 

5 wherein the one or more light sources of the 

position lighting device fixed to the first mock shoot- 
ing device emit light with a first characteristic and 
the one or more light sources of the position lighting 
device fixed to the second mock shooting device 

io emit light with a second characteristic and wherein 
the first characteristic and the second characteristic 
are different. 

9. An apparatus comprising 
15 a computing device; 

a screen device; 

a mock shooting device is comprised of a lighting 
device containing one or more light sources which 
are fixed to the mock shooting device; and 

20 wherein one or more light sources of the light- 

ing device emit invisible light; 
and wherein the computing device uses the light 
from the one or more light sources of the lighting 
device to determine whether the mock shooting de- 

25 vice is aimed towards a location on the screen de- 
vice. 

10. The apparatus of claim 9 further comprising 

a video camera that captures video images of the 
30 one or more light sources of the lighting device; and 
wherein one or more light sources of the light- 
ing device emit light which is invisible to human 
eyes but visible to the video camera; and 

wherein the video camera is electrically con- 
35 nected to a computing device and the video camera 
provides data about the one or more light sources 
of the lighting device to the computing device. 

11. An apparatus comprising 
40 a computing device; 

a screen device; 

a mock shooting device which can send a signal via 
a communications line; 

and a lighting device containing one or more light 
45 sources which are fixed to or as a built-in part of the 
mock shooting device; and 

wherein one or more light sources of the light- 
ing device emit light; and 

wherein the computing device uses the light 
so from the one or more light sources of the lighting 
device to determine whether the mock shooting de- 
vice is aimed towards a location on the screen de- 
vice; 

and wherein the communications line is used to 
55 send one or more command signals from the mock 
shooting device to the computing device. 

12. The apparatus of claim 11 further comprising 
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a video camera that captures video images of the 
one or more light sources of the lighting device; and 
wherein the video camera is electrically con- 
nected to a computing device and the video camera 
provides data about the one or more light sources 
of the lighting device to the computing device. 
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FIG. 3 
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